Unpacking Codebases with Generative AI

A LangGraph + Neo4j + Streamlit Project for Codebase Analysis and Interactive Querying

Introduction

Project Overview and Purpose

Every developer has faced the challenge of diving into an unfamiliar codebase - trying to untangle dependencies, understand architectural decisions, and figure out “how does this all work together?” For our class project, we set out to address this universal pain point by creating a Code Analysis Agent tool that makes sense of complex Python codebases through the power of AI. By combining language models with graph databases, we’ve developed a system that can “read” code the way humans do, identifying relationships between components and allowing developers to ask natural questions about structure and functionality.

The Code Analysis Agent functions as an intelligent code exploration assistant, transforming source code into queryable knowledge graphs. By applying natural language processing to code elements, it creates semantic representations that capture the architecture, dependencies, and functional relationships within a codebase. This approach enables developers to gain insights through conversational queries rather than manual code traversal, significantly reducing the cognitive load of understanding unfamiliar or complex projects.

Key Features

The Code Analysis Agent offers several core capabilities:

• It automatically analyzes Python codebases by parsing files and generating graphs that capture semantic relationships between modules, classes, and functions.
• It employs a multi-agent reasoning architecture built with LangGraph, enabling dynamic selection between micro-level (fine-grained) and macro-level (high-level) analysis workflows.
• It provides interactive visualizations, including relationship graphs and mermaid diagrams, to present architectural information in a digestible format.
• It allows users to query the codebase using natural language, returning LLM-generated explanations, semantic paths, or visual summaries depending on the request.

These features combine to offer both depth and flexibility, making code exploration faster, more structured, and more accessible.

Technologies Used

The system integrates several modern AI and graph-based technologies:

• LangGraph is used to build the agentic workflow, allowing the system to dynamically route user queries to the appropriate analysis tools.
• Neo4j serves as the backend graph database, storing semantic relationships extracted from the codebase.
• Streamlit provides a lightweight, interactive web interface that guides users through repository ingestion, analysis, and querying.
• OpenAI GPT-4.1-mini acts as the primary language model for natural language understanding, Cypher query generation, and RAG-based retrieval.

Together, these technologies enable the Code Analysis Agent to operate at both a structural and semantic level, offering users a unified and intelligent code exploration experience.

System Architecture

Overall Workflow

The Code Analysis Agent follows a multi-stage workflow that transforms raw code repositories into queryable knowledge graphs:

1. Repository Ingestion: When a user provides a GitHub repository URL through the Streamlit interface, the system clones it locally and presents the file structure. Users can then select specific source folders for analysis.

2. Graph Construction: Selected Python files are processed through a pipeline that:

• Parses each file to extract classes, functions, and their relationships
• Uses LLMs to generate natural language descriptions of code components
• Structures these descriptions into a graph format with typed nodes and edges
• Uploads the resulting graph to a Neo4j database

3. Query Processing: Once the graph database is populated, users can submit natural language questions about the codebase. These queries are:

• Routed to the appropriate agent type via the supervisor
• Translated into appropriate tool calls (Cypher queries, RAG lookups, etc.)
• Processed to generate responses and/or visualizations
• Presented to the user via the Streamlit interface

This workflow enables a seamless transition from raw code to interactive analysis without requiring users to understand the underlying graph structures or query languages.

Agent Router System

The heart of the system’s intelligence is its agent router architecture, implemented using LangGraph. This architecture enables dynamic, state-driven processing of user queries through specialized agents:

The agent router consists of three primary components:

1. Supervisor Node: Analyzes user queries to determine whether they require detailed code relationship analysis (micro) or high-level architectural summaries (macro). This node implements a decision-making process based on the query content and current conversation state.

2. Micro Agent: Specialized for detailed code analysis, this agent has access to tools for:

• Building and executing Cypher queries against the Neo4j database
• Retrieving specific relationship information between code components
• Generating network visualizations of code relationships
• Providing detailed explanations using RAG-based retrieval

3. Macro Agent: Focused on high-level architectural understanding, this agent can:

• Generate natural language summaries of the codebase structure
• Create Mermaid diagrams showing logical component relationships
• Provide strategic insights about code organization

Each agent operates within a standardized LangGraph workflow, with structured prompting that guides the LLM toward appropriate tool selection and reasoning.

System Diagrams

Data Flow Diagram

Data Flow Diagram

The Data Flow Diagram illustrates the comprehensive pipeline through which code is processed, analyzed, and made available for interactive querying. At the foundation, Git repositories provide source code that enters our processing pipeline through a multi-stage transformation:

First, the code parser extracts structural elements from Python files, which are then enriched by the LLM processor that generates semantic descriptions and relationships. This processed information flows into two critical storage components: the Neo4j graph database (storing structured relationships) and the RAG vectorstore (enabling semantic retrieval).

The central LangGraph Router orchestrates the system’s intelligence, dynamically directing queries to appropriate analysis tools. These tools are divided into micro and macro capabilities - micro tools leverage the Cypher Query Builder to extract precise relationship data from Neo4j, while macro tools generate high-level summaries and architectural diagrams. Both tool sets produce visualizations and textual outputs that are presented through the Streamlit interface.

This architecture enables bidirectional information flow, where user queries trigger specific analysis pathways and results are transformed into appropriate visualizations or textual responses. The color-coded components highlight the distinct functional areas: orange for user-facing interfaces, blue for agent routing logic, and neutral for data processing and storage elements.

Component Diagram

Component Diagram

The Component Diagram provides a structural view of the system’s modular architecture, highlighting the key functional units and their interactions. The system is organized into three primary component groups:

The Streamlit App serves as the user-facing layer, containing the Repo Cloner for importing code repositories, the Chat Interface for query input and response display, and the Visualization Panel for rendering interactive graphs and diagrams. This frontend communicates bidirectionally with the LangGraph Router, which acts as the central orchestration mechanism.

The Router intelligently delegates user queries to two specialized tool sets: Micro Tools and Macro Tools. Micro Tools include the Cypher Generator for constructing database queries, the Visualizer for rendering relationship graphs, and the RAG Query Tool for semantic retrieval. Macro Tools consist of the Text Generator for producing high-level summaries and the Mermaid Generator for creating architectural diagrams.

The Database layer forms the foundation of the system, housing both the Neo4j graph database for structured relationships and the RAG Vectorstore for semantic embeddings. This layer also includes the Database Initializer for setup and Graph Builders for transforming code into graph structures.

This component architecture emphasizes clean separation of concerns while maintaining efficient communication pathways between modules. The color-coding distinguishes user interface components (orange) from agent and tool components (blue) and data storage components (neutral), reflecting their distinct roles in the overall system.

Technical Implementation

Initialize Database

When a user submits a GitHub repository through the frontend, a Langchain InMemory Vectorstore and a Neo4j instance are spun up, ready to store data as soon as preprocessing wraps up.

Data Preprocessing

The raw code from the repository is first processed into natural language to make it easier to work with. From there, the data is inserted into both the graph database and the vectorstore. Different strategies are used to generate the text for each system, tailoring the structure and content to match the specific role of each database — ensuring the graph database captures meaningful relationships, while the vectorstore is optimized for efficient retrieval and semantic understanding.

GraphDB

The source code is converted into natural language descriptions suitable for building a GraphRAG system, with specific attention given to clearly specifying the type of each node and including the full path.

This step is critical because graph transformers, when processing raw code directly, do not inherently understand that class names represent classes, or that function names represent functions. For example, a function named remove_duplicates might be mistakenly treated as a general “concept” rather than correctly recognized as a function, unless the natural language description explicitly states it.

Each module is processed separately to ensure that all fine-grained details are captured. Handling modules individually is important because longer text inputs reduce the ability of the transformer to detect meaningful relationships between entities.

However, processing modules in isolation makes it harder to identify connections between modules, such as imports and cross-references. To solve this, the full names—including the module paths—must be preserved during translation.

An example of the natural language translation is:

deduplication.bloom_filter.BloomFilter is a class defined in the module deduplication.bloom_filter.

After these natural language descriptions are created, a graph transformer is used to generate nodes and relationships, which are then inserted into a Neo4j database.

VectorDB

The preprocessing required for building the vector database is relatively lightweight compared to the more involved process of generating data for the graph database. To prepare the content for the vector store, code snippets are first passed through a translation step, where they are converted into detailed natural language explanations. This process is similar to submitting raw code to a large language model and asking it to produce a thorough, step-by-step breakdown of the code’s structure and behavior. Once translated, the explanations are concatenated into larger blocks of text, segmented into smaller, manageable chunks to optimize downstream retrieval. These chunks are then embedded and stored in an in-memory vector database, enabling fast and efficient semantic search.

Agentic Workflow (Process User Input)

The supervisor agent manages the conversation flow by deciding the next step based on the current state. It prepares the input by adding a system prompt to the conversation history, providing context for the language model. It then asks the model for a structured output, which suggests whether to move to a “macro,” “micro,” or “finish” state. If the model responds with “FINISH,” the agent maps it to an internal END marker for consistency.

After making the decision, the supervisor generates an AI message explaining the chosen route and appends it to the conversation history. Finally, it returns a Command object that tells the system where to go next and includes the updated messages. In short, the supervisor reads the conversation, picks the next action, and records its reasoning clearly for the user or system to see.

Micro Agent

The purpose of the micro-agent is to assist end users in understanding the finer details of the codebase. Users can ask questions about modules, classes, methods, functions, and packages. What sets this agent apart is its ability to provide a full answer by performing both GraphRAG and RAG for a single answer. It traces functional connections by generating Cypher queries and retrieving insights from a graph database, and using these relationships as a foundation, the agent can further enhance its responses either by performing a vector search on a vector store—containing natural language explanations of the code—or by generating visualizations of the functional connections.

The micro-agent is equipped with three key tools, each serving a distinct role:

1. Cypher Query Chain (GraphRAG)
   • Input: User request
   • Process: Generates a Cypher query based on the user’s input (received from the frontend)
   • Output1: Retrieves the relevant relationships and nodes from the graph database

        Example)
        {
            'path': 
            [{'id': 'Utils.Utils'}, 
            'IMPORTED_FROM', 
            {'id': 'Utils.Utils.Unionfind'}, 
            'DEFINES',
            {'id': 'Utils.Utils'}, 
            'IMPORTED_FROM', 
            {'id': 'Utils.Utils.Minhash'}, 
            'USES', 
            {'id': 'Utils.Use_Cases.Nearest_Neighbor_Search'},
            'IMPORTED_IN', 
            {'id': 'Deduplication.__Main__'}]
        }

   • Output2: Converts the retrieved relationships into clear, natural language explanations

        Example)
        `Utils.Utils` imports `Utils.Utils.Minhash`, 
        which is used in `Utils.Use_Cases.Nearest_Neighbor_Search` 
        and is imported into `Deduplication.__Main__`

2. Text Generation (RAG)
   • Input: Output2 from the Cypher Query Chain tool
   • Process: Searches the vector database for semantically similar documents and enriches the initial relationship explanations with broader context
   • Output: A fully developed text response that describes both the relationships and the functionality in depth

        Example)
        The `Utils.Utils` module plays a crucial role in the `Deduplication.__Main__` module by providing fundamental utilities that aid in document deduplication and nearest neighbor search operations. Here's an overview of their relationship:
   
        1. **Union-Find Operations**: The `UnionFind` class in `Utils.Utils` offers efficient set operations important for managing clusters of duplicates and merging them, which is essential for deduplication.

3. Visualization Generation
   • Input: Output1 from the Cypher Query Chain tool
   • Process: Uses the retrieved graph data to create visual representations
   • Output: A final visualization that illustrates the relationships between nodes, helping users quickly grasp the structure and connections within the code

Macro Agent

The macro agent is designed to give users a high-level understanding of the entire codebase. Instead of relying on a traditional Retrieval-Augmented Generation (RAG) approach, it uses a Context-Augmented Generation (CAG) pipeline. This method allows the agent to deliver a more coherent and contextually rich summary in a single, streamlined response. By having access to the full codebase along with its structural layout, the macro agent can generate summaries that not only capture individual components but also highlight how they fit together across the repository.

The macro agent is equipped with two primary tools, each serving a distinct purpose:

1. Summary text generator
   • Input: User request + entire codebase + repository tree
   • Process: prompt engineered to provide text suitable for mermaid graphs
   • Output: Summary of entire codebase

        Example)
        #### Main Modules & Purposes:
        1. **deduplication/**
        - **bloom_filter.py**: Bloom Filter implementation for efficient set membership checks.
        - **dedup.py**: Baseline deduplication using hash comparisons and token overlap.
        - **LSH.py**: Locality Sensitive Hashing with MinHash for duplicate detection.
        - **LSHForest.py**: Uses multiple trees for robustness in LSH-based detection.
        - **LSHImproved.py**: Multi-probe techniques in LSH to improve efficiency.

2. Mermaid graph generator
   • Input: Output of the summary text generator tool
   • Process: Natural language is translated to mermaid graph.
   • Output: Mermaid graph generated of funcitonalities of the codebase.

Technical Decision Rationale

1. Why Neo4j over Memgraph provided by LangChain?
   Initially, we considered using Memgraph’s in-memory graph database for the pipeline. However, during testing, we discovered that Memgraph did not fully support the latest Cypher query syntax, particularly when working with a Cypher query generator. This limitation led us to choose Neo4j, which reliably supports the most up-to-date Cypher grammar and provides a robust foundation for our graph-based analysis.

2. Why GPT models over Claude?
   While Claude models are often praised for their code generation capabilities, we found that GPT models excelled in linguistic tasks, particularly in retrieving nodes and relationships for graph construction. Their superior performance in understanding and processing natural language made them the ideal choice for our system.

3. Why GPT-4.1 Mini over GPT-4.0?
   To optimize the data preprocessing and loading pipeline, we parallelized API calls to improve efficiency. However, GPT-4.0’s token-per-minute (TPM) limit of 30,000 significantly restricted our ability to make multiple concurrent calls. GPT-4.1 Mini, with its much higher TPM limit of 300,000, allowed us to maintain the advantages of GPT models while achieving a substantial performance boost.

4. Why a manual Cypher RAG chain instead of LangChain’s pre-built GraphCypherQAChain?
   We opted to build a custom pipeline for generating, executing, and retrieving Cypher queries rather than using LangChain’s pre-built GraphCypherQAChain. This decision was driven by the need for greater customization, such as modifying system prompts and providing specific examples to enforce a strict format for Cypher queries. This tailored approach ensured that our pipeline could handle the unique requirements of our graph-based system effectively.

Usage Guide

Installation

The Code Analysis Agent can be run either locally or via Docker.

Option 1: Local Installation

1. Install Python dependencies:

pip install -r requirements.txt

2. Create a .env file with your environment variables:

NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=your_password
OPENAI_API_KEY=your_openai_key

3. Start the Streamlit app:

streamlit run src/streamlit_app.py

Option 2: Docker Deployment

1. Ensure Docker and Docker Compose are installed.

2. Set environment variables as above.

3. Set up virtual environment:

uv venv
uv sync --locked

4. Build and start containers:

docker-compose up -d

5. Access the app at http://localhost:8501.

Note: Requires a running Neo4j instance. Optimized for Python 3.10+.

---

Basic Usage

After launching the application:

• Clone a Repository: Enter a GitHub repository URL and click “Clone Repository.”
• Select Source Folders: Choose folders you want to analyze (e.g., src, backend/api, .).
• Initialize the Database: Start the analysis pipeline to extract code structure and relationships.
• Ask Questions: Use natural language queries to explore code relationships and architecture.
• Visualize: Generate interactive graphs and downloadable diagrams of the codebase.

Example Queries

To ensure the agent provides an accurate response, the user must specify the exact locations of the nodes. Different prompts activate different pipelines within the agent. Below are example prompts, using a sample repository, to illustrate how to get the response you want.

• Example Repository: https://github.com/psf/requests.git

• Micro Agent with Text (GraphRAG + RAG): How is the Requests.Help.Requests_Version related to the Requests.Init module?

• Micro Agent with Text (RAG): What does the Requests.Auth.Httpdigestauth class do?

• Micro Agent with Visualization (GraphRAG): Give me a visualization of how the Requests.Exceptions module is related to the Requests.Api module.

• Macro Agent: Give me a high-level overview of the codebase.

Demonstration

For a full demonstration of the system in action — including repository ingestion, graph construction, querying, and visualization — please watch our 5-minute demo video:

YouTube Demo

The video covers:

• Setting up the application
• Cloning and analyzing a repository
• Interacting with both the micro and macro agents
• Viewing and downloading visualizations
• Example natural language queries

Evaluation

Evaluating the pipeline for querying a Graph Database, enriching context through a vector database, or using a CAG pipeline for summarizing a codebase proved to be a complex task. As a result, our evaluation primarily focused on the performance of RAG and the overall system efficiency.

RAG Evaluation

To assess the performance of our Retrieval-Augmented Generation (RAG) pipeline, we employed a combination of RAGAS and LLM as a judge to incorporate various aspects that one model may be missing out on. The definition of the two methods of evaluation methods are as follows :

Synthetic Query Evaluation with RAGAS

Using the RAGAS framework, we generate synthetic queries tailored to the codebase and evaluated the system’s ability to retrieve and synthesize relevant information. This approach allowed us to systematically measure the precision and recall of the RAG pipeline, ensuring that the retrieved context aligns with user intent.

LLM as a judge

Leveraging a large language model (LLM) as an impartial evaluator to judge the quality of responses. The LLM assessed the outputs based on criteria such as factual correctness, contextual relevance, and linguistic clarity. This method provided an additional layer of validation, ensuring that the system’s responses met high standards of quality and usability.

Results

The context recall by RAGAS achieved an average score of 0.8, indicating that the system consistently retrieved relevant information. However, factual correctness ranged between 0.3 and 0.4, suggesting that while the correct context was retrieved, the generated responses often contained inaccuracies. Using the default F1 score in RAGAS to measure factual correctness, we observed that retrieving 10 related documents introduced additional information, which diluted precision and lowered the overall F1 score compared to the synthetic responses generated by RAGAS.

To address this, we switched to using recall as the primary metric, which better accounted for the differences, resulting in an improved factual correctness score of approximately 0.6. While this was an improvement, it still fell short of expectations. To investigate further, we analyzed a specific example where the factual correctness score was 0.3. Upon human evaluation, the response appeared accurate. We then used an LLM as a judge to assess the similarity between the synthetic and pipeline-generated responses, which yielded a high similarity score of 0.9.

This analysis demonstrated that our RAG pipeline is robust and capable of producing accurate answers, even if the default evaluation metrics do not fully reflect its performance.

Efficiency

Efficiency is critical for making the Code Analysis Agent usable on real-world codebases. In this section, we evaluate two major aspects: the time required for data ingestion (preprocessing and database population) and the latency experienced during interactive query responses.

Data Preprocessing/Ingestion

The data preprocessing pipeline consists of several sequential stages: transforming raw Python files into natural language descriptions, generating graph structures, and uploading the results into the Neo4j and vector databases. The following table summarizes the time taken for each of these stages during a typical run on a moderately sized codebase:

Stage	Time (Graph DB)	Time (Vector DB)
Code → NL	71s	57s
Graph Generation	381s	X
Database Input	62s	0s
Total Time	8m 34s	57s

These timings highlight that the initial overhead for ingestion is non-trivial, especially due to the time-intensive nature of code parsing and language model processing. However, once ingestion is complete, the system is able to serve user queries efficiently without needing to reprocess the underlying codebase.

Note

Note: Future improvements such as incremental graph updates (see Future Extensions) could reduce ingestion times significantly.

Response Latency

Response time is crucial for user experience when interacting with the system. We separately measure the latency for both micro-agent and macro-agent workflows, distinguishing between responses that generate natural language answers and those that involve visualization generation.

Micro Agent

The micro agent handles fine-grained codebase queries, such as retrieving detailed relationships between modules or classes. Latency is broken down into the following stages:

Stage (Text Response)	Time (Text Response)	Stage (Visualization Response)	Time (Visualization Response)
Generate/Execute Query	140s	Generate/Execute Query	140s
RAG Retrieval	10s	Create Visualization	42s
Total Time	150s	Total Time	182s

Micro-agent responses typically involve a multi-step process: building a Cypher query, executing it against the Neo4j database, optionally performing retrieval-augmented generation (RAG) for additional explanation, and formatting the final output. Visualization workflows replace RAG retrieval with the generation of relationship graphs.

Macro Agent

The macro agent generates high-level overviews of the codebase structure and creates mermaid diagrams for architectural summaries. Response timing for macro workflows is shown below:

Stage	Time
LLM Summarization	16s
Mermaid Diagram Generation	33s
Total Time	49s

In contrast to the micro agent, the macro agent operates primarily through direct text generation followed by conversion into visual diagrams. As a result, its overall latency tends to be more predictable but may still vary depending on the size and complexity of the input codebase.

Overall, while response times are acceptable for typical use cases, further optimizations could enhance responsiveness, particularly for large-scale repositories or complex relational queries.

Future Work and Enhancements

While the Code Analysis Agent demonstrates strong functionality in its current form, several important extensions and improvements have been identified for future development:

Efficiency and Scalability

Processing speed, especially during graph construction and query response generation, remains a bottleneck. Future work should explore optimization strategies, such as:

• Implementing more aggressive parallelization during code parsing and graph uploading.
• Utilizing more efficient data structures for representing intermediate graph information.
• Investigating incremental updates to the Neo4j database rather than full re-uploads.

These changes could significantly reduce initialization times and improve system responsiveness for larger repositories.

Smarter Entity Matching

At present, the system requires strict, fully-qualified names (e.g., src.app.graph) to correctly identify modules and functions. A more robust approach would involve:

• Implementing flexible matching techniques that map user queries phrased in natural language (e.g., “the graph module”) to underlying code elements.
• Leveraging LLM-based entity resolution to handle ambiguous references, especially when names are shared across different parts of the codebase.

This enhancement would make the agent feel more conversational and forgiving, particularly in exploratory settings.

Improved Query Precision

The current Cypher-based graph queries sometimes yield noisy or overly broad results, connecting modules through weak or incidental relationships (such as shared use of common libraries like numpy). Future improvements could focus on:

• Defining stronger notions of “semantic relevance” between nodes.
• Incorporating path scoring heuristics to prioritize meaningful architectural links over incidental ones.
• Introducing adjustable query parameters (e.g., “strict mode”) to allow users to control the breadth of relationships retrieved.

This would enable more targeted, insightful exploration of code structures.

Enhanced Agent Collaboration

Currently, the agent router selects either the micro or macro agent exclusively for a given query. Future work could improve the system by:

• Allowing micro- and macro-agents to communicate or pass intermediate results.
• Enabling workflows where high-level summaries from the macro agent inform detailed Cypher queries from the micro agent (or vice versa).
• Building hierarchical reasoning chains that integrate structural overviews with fine-grained relational data.

Such enhancements would allow richer, multi-layered responses, providing users with both strategic overviews and precise technical details in a seamless experience.

Conclusion

This project set out to address a fundamental challenge faced by developers: understanding unfamiliar codebases. By integrating large language models, graph databases, and agentic workflows, we created a Code Analysis Agent capable of transforming raw code into a structured, queryable, and visualizable knowledge base.

Throughout development, we emphasized both depth and flexibility. The system enables detailed exploration of functional relationships via micro-level agents, while also supporting strategic overviews through macro-level summaries and visualizations. By dynamically routing queries to the appropriate agent, the LangGraph-based architecture allows for nuanced, multi-faceted interactions with complex repositories.

Our technical choices, such as favoring Neo4j over Memgraph, GPT-4.1-mini over other LLM options, and custom GraphRAG pipelines over prebuilt solutions, were driven by a desire for precision, control, and future scalability. Extensive preprocessing ensured that both graph and vector representations of the codebase captured meaningful, semantically rich information, enabling a smooth user experience during querying.

While the system achieved its primary goals, evaluation highlighted several areas for future improvement, including efficiency bottlenecks, smarter entity resolution, and more cohesive micro/macro agent coordination. Addressing these challenges could further enhance the system’s usability and scalability, making it suitable for larger and more diverse real-world codebases.

Ultimately, the Code Analysis Agent demonstrates how modern AI tools can significantly reduce the cognitive burden of code comprehension. By turning static source code into a dynamic, interactive knowledge system, this project lays the groundwork for more intelligent, accessible, and intuitive approaches to understanding and navigating software architecture.